The Internet is now being use to “broadcast” or carry continuous video and audio program streams. But the nature of the Internet is that packets of data are individually launched toward the recipient, without any guarantee of delivery time or delivery at all. So long data streams can experience problems at the receiving end because they must be broken up into packets. Such packets can be corrupted, late in arriving, or missing altogether.
The type of streaming media involved will determine how sensitive the content is to these transmission defects. Some error detection and recovery schemes can be used to detect and fill in some of the holes in the received data streams. The Internet itself between a particular point-A and point-B can introduce communication problems on the data streams too.
Various Internet users need to know how well the Internet is performing and how well the content is surviving the transmission. Sometimes this information is needed before actually sending a program or requesting it, and sometimes the need-to-know is post-transmission. For example, broadcasters and Internet Service Providers (ISP's) need to assess actual delivery quality of the program to the consumer.
Absent requiring manual feedback from the user, the prior art does not do a good job of assessing streaming media quality from the end-user's perspective. In conventional systems, the data collection, test setup, and data analysis tasks are not fully automated. An automated tool is needed that is compatible with commercial client-side media player applications and that evaluates the actual media streams in use.
Conventional software measures and assesses streaming media quality from the transmission source, or some mid-point along the way to the consumer. The prior art does not generally provide open interfaces or linkage to other management and assessment systems. For example, NetIQ Corporation's CHARIOT product uses specialized endpoints that send and receive synthetic traffic to emulate real media servers and receivers, in packet size and rate. See, www.netiq.com. But such tool cannot assess stream start-up delay, player stall, or other characteristics relevant to actual media clients. It is not scalable, and it cannot interpret or adequately aggregate data for high-level representations of stream quality.
BroadStream (Woodland Hills, Calif.) markets audience metrics, performance assurance, diversified delivery networks, media services, and professional services. See, www.broadstream.com. A software application of theirs monitors the quality of streaming media and can adjust it on-the-fly if the software detects a problem. If one delivery network fails, the stream can be re-routed without significantly affecting the end user. The software generates a report regarding the end-user's experience and behavior during the Webcast. The BroadStream software reports on how well a Webcast transmission is being received, who's watching it, and for how long.
The audience metrics service assesses the content quality at its destination and how many people received it. The BroadStream.com service downloads a JAVA applet to the client that looks at actual end-user activity and data quality, and reports a quality assessment back to the content producers. But such requires end-user involvement. It does not do independent testing or fault isolation, and it requires clients to communicate to a single central service location on the Internet. No authentication mechanism or control by the end user is provided to disable the assessment and the associated information collection.
Streamcheck (Toronto, Ontario, Canada) provides services that operate similar to those of Broadstream. See, www.streamcheck.com. A subjective measurement is generated from measurements of the initial connection and total stream buffering time, as compared to total playback time. Streamcheck does only scheduled tests on pre-submitted media streams and uses synthetic test clients.
Keynote Systems, Inc., (San Mateo, Calif.) provides products that measure the streaming media quality at locations along the Internet backbone using actual client applications. STREAMING PERSPECTIVE is a performance measurement service for measuring the quality of streaming audio and video on the Internet. The service is a completely outsourced way to measure, assure, compare, diagnose and improve the performance of streaming content. See, www.keynote.com. However, such is limited to test-mode functionality only. They do not measure desktop performance as seen by the actual end-users.
Lariat Software (Sunnyvale, Calif.) markets their MEDIA REPORTS product that includes network and server reports focused on packet delivery statistics, delivery efficiency, server errors and files causing server errors, server capacity statistics. Such information is used to understand how servers are delivering content, and to spot delivery problems and file errors. These reports provide server usage patterns for maintenance scheduling and analyzing needs for infrastructure server and licenses investment. See, www.lariat.com/mediarep/index.htm. MEDIA REPORTS gets information from the log files of media servers to provide an assessment of stream quality. Such techniques are proprietary, and are not open to general integration into a complete assessment service. They do not provide for testing and fault isolation independent of end-user cooperation, and cannot accurately assess quality as seen at the client.
Real-time transport protocol (RTP) for media streaming is related to real-time control protocol (RTCP) that reports on the quality of received streams from the client. See, H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson, “RTP: A Transport Protocol for Real-Time Applications,” IETF RFC 1889, January 1996. The RTCP protocol is not very scalable, and is generally disabled in real production deployments of the media servers and players that include standard RTP. The commercially popular servers and players do not fully support standard RTP.
RealNetworks' REALPLAYER (www.real.com), Microsoft's WINDOWS MEDIA PLAYER, and other proprietary media players, support quality feedback agents that attempt to adjust stream server characteristics. Generally, such agents work with the server to enable lower bit-rate media streams, and are not provided for general fault diagnosis.
Recent prior art has focused on instrumenting client-side media players in order to assess the quality of received video streams. For example, Y. Wang, M. Claypool, and Z. Zuo, published “An Empirical Study of RealVideo Performance Across the Internet,” Proceedings of the ACM SIGCOMM Internet Measurement Workshop 2001, San Francisco, Calif., November 2001, that studies user-perceived video quality using an instrumented version of RealPlayer. The application is fully user-controlled, but it requires the user to manually rate their perception of the video quality once a clip has played.
D. Loguinov and H. Radha, describe in “Measurement Study of Low-Bitrate Internet Video Streaming,” Proceedings of the ACM SIGCOMM Internet Measurement Workshop 2001, San Francisco, Calif., November 2001, a measurement study of streaming video to dial-up modem clients that uses instrumented MPEG-4 receivers. The video client used measured network/path and packet statistics without mapping such metrics to the received video quality as perceived by the user.
The mapping between objective measurements and subjective video quality for MPEG-1 and Windows Media clips over a DiffServ network is described by W. Ashmawi, R. Guerin, S. Wolf, and M. H. Pinson, in “On the Impact of Policing and Rate Guarantees in Diff-Serv Networks: A Video Streaming Application Perspective,” Proceedings of the ACM SIGCOMM'01, San Diego, Calif., August 2001, pp. 83–95. The mapping measures both the sent and received video streams to extract key parameters that are then compared in building a subjective quality score. The software depends on knowing about both the sent and received video streams. The quality score calculation is done off-line, and not in real-time as the stream is playing.